The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

缺乏对深度学习系统的洞察力阻碍了他们的系统设计。在科学和工程学中，建模是一种用于了解内部过程不透明的复杂系统的方法。建模用更简单的代理代替复杂的系统，该系统更适合解释。从中汲取灵感，我们使用高斯流程为神经网络构建了一类代理模型。我们没有从神经网络的某些限制案例中得出内核，而是从经验上从神经网络的自然主义行为中学习了高斯过程的内核。我们首先通过两项案例研究评估我们的方法，灵感来自先前对神经网络行为的理论研究，在这些案例研究中，我们捕获了学习低频的神经网络偏好，并确定了深层神经网络中的病理行为。在进一步的实践案例研究中，我们使用学识渊博的内核来预测神经网络的泛化特性。

尽管深度学习预测模型在歧视不同阶层方面已经成功，但它们通常会遭受跨越包括医疗保健在内的具有挑战性领域的校准不良。此外，长尾分布在深度学习分类问题（包括临床疾病预测）中构成了巨大挑战。最近提出了一些方法来校准计算机视觉中的深入预测，但是没有发现代表模型如何在不同挑战性的环境中起作用。在本文中，我们通过对四个高影响力校准模型的比较研究来弥合从计算机视觉到医学成像的置信度校准。我们的研究是在不同的情况下进行的（自然图像分类和肺癌风险估计），包括在平衡与不平衡训练集以及计算机视觉与医学成像中进行。我们的结果支持关键发现：（1）我们获得了新的结​​论，这些结论未在不同的学习环境中进行研究，例如，结合两个校准模型，这些模型都可以减轻过度启发的预测，从而导致了不足的预测，并且来自计算机视觉模型的更简单的校准模型域往往更容易被医学成像化。 （2）我们强调了一般计算机视觉任务和医学成像预测之间的差距，例如，校准方法是通用计算机视觉任务的理想选择，实际上可能会损坏医学成像预测的校准。 （3）我们还加强了自然图像分类设置的先前结论。我们认为，这项研究的优点可以指导读者选择校准模型，并了解一般计算机视觉和医学成像域之间的差距。

语言模型既展示了定量的改进，又展示了新的定性功能，随着规模的增加。尽管它们具有潜在的变革性影响，但这些新能力的特征却很差。为了为未来的研究提供信息，为破坏性的新模型能力做准备，并改善社会有害的效果，至关重要的是，我们必须了解目前和近乎未来的能力和语言模型的局限性。为了应对这一挑战，我们介绍了超越模仿游戏基准（Big Bench）。 Big Bench目前由204个任务组成，由132家机构的442位作者贡献。任务主题是多样的，从语言学，儿童发展，数学，常识性推理，生物学，物理学，社会偏见，软件开发等等。 Big-Bench专注于被认为超出当前语言模型的功能的任务。我们评估了OpenAI的GPT型号，Google内部密集变压器体系结构和大型基础上的开关稀疏变压器的行为，跨越了数百万到数十亿个参数。此外，一个人类专家评估者团队执行了所有任务，以提供强大的基准。研究结果包括：模型性能和校准都随规模改善，但绝对的术语（以及与评估者的性能相比）；在模型类中的性能非常相似，尽管带有稀疏性。逐渐和预测的任务通常涉及大量知识或记忆成分，而在临界规模上表现出“突破性”行为的任务通常涉及多个步骤或组成部分或脆性指标；社交偏见通常会随着含糊不清的环境而随着规模而增加，但这可以通过提示来改善。

人工智能（AI）系统在接下来的几十年中有很大的希望可以改善医疗保健。具体而言，利用多个数据源和输入模式的AI系统有望成为一种可行的方法，可以在广泛的应用程序中提供更准确的结果和可部署的管道。在这项工作中，我们提出并评估一个统一的医学中的整体AI（HAIM）框架，以促进利用多模式输入的AI系统的生成和测试。我们的方法使用可通用的数据预处理和机器学习建模阶段，可以很容易地适应医疗保健环境中的研究和部署。我们通过训练和表征基于MIMIC-IV-MM的14,324个独立模型来评估我们的HAIM框架，该模型是一种多模式临床数据库（n = 34,537个样本），其中包含7,279个独特的住院和6,485名患者，涵盖了4个数据模态的所有可能输入组合（即，所有可能的输入组合）表格，时间序列，文本和图像），11个独特的数据源和12个预测任务。我们表明，该框架可以始终如一地生产出在各种医疗保健示范中超过相似的单源方法的模型（乘以6-33％），包括10种不同的胸部病理学诊断，以及休息时间和48小时的死亡率预测。我们还使用Shapley值量化了每种模式和数据源的贡献，这证明了数据类型重要性的异质性以及在不同医疗保健相关的任务中多模式输入的必要性。我们的整体医学AI（HAIM）框架的可推广性能和灵活性可以为未来的临床和运营医疗环境中的多模式预测系统提供有希望的途径。

制定了具有机器学习模拟（骆驼）项目的宇宙学和天体物理学，通过数千名宇宙的流体动力模拟和机器学习将宇宙学与天体物理学结合起来。骆驼包含4,233个宇宙学仿真，2,049个n-body和2,184个最先进的流体动力模拟，在参数空间中采样巨大的体积。在本文中，我们介绍了骆驼公共数据发布，描述了骆驼模拟的特性和由它们产生的各种数据产品，包括光环，次麦，银河系和空隙目录，功率谱，Bispectra，Lyman - $ \ Alpha $光谱，概率分布函数，光环径向轮廓和X射线光子列表。我们还释放了超过骆驼 - 山姆的数十亿个星系的目录：与Santa Cruz半分析模型相结合的大量N身体模拟。我们释放包含350多个Terabytes的所有数据，并包含143,922个快照，数百万光环，星系和摘要统计数据。我们提供有关如何访问，下载，读取和处理数据AT \ URL {https://camels.readthedocs.io}的进一步技术详细信息。

As Artificial and Robotic Systems are increasingly deployed and relied upon for real-world applications, it is important that they exhibit the ability to continually learn and adapt in dynamically-changing environments, becoming Lifelong Learning Machines. Continual/lifelong learning (LL) involves minimizing catastrophic forgetting of old tasks while maximizing a model's capability to learn new tasks. This paper addresses the challenging lifelong reinforcement learning (L2RL) setting. Pushing the state-of-the-art forward in L2RL and making L2RL useful for practical applications requires more than developing individual L2RL algorithms; it requires making progress at the systems-level, especially research into the non-trivial problem of how to integrate multiple L2RL algorithms into a common framework. In this paper, we introduce the Lifelong Reinforcement Learning Components Framework (L2RLCF), which standardizes L2RL systems and assimilates different continual learning components (each addressing different aspects of the lifelong learning problem) into a unified system. As an instantiation of L2RLCF, we develop a standard API allowing easy integration of novel lifelong learning components. We describe a case study that demonstrates how multiple independently-developed LL components can be integrated into a single realized system. We also introduce an evaluation environment in order to measure the effect of combining various system components. Our evaluation environment employs different LL scenarios (sequences of tasks) consisting of Starcraft-2 minigames and allows for the fair, comprehensive, and quantitative comparison of different combinations of components within a challenging common evaluation environment.

Reinforcement-learning agents seek to maximize a reward signal through environmental interactions. As humans, our contribution to the learning process is through designing the reward function. Like programmers, we have a behavior in mind and have to translate it into a formal specification, namely rewards. In this work, we consider the reward-design problem in tasks formulated as reaching desirable states and avoiding undesirable states. To start, we propose a strict partial ordering of the policy space. We prefer policies that reach the good states faster and with higher probability while avoiding the bad states longer. Next, we propose an environment-independent tiered reward structure and show it is guaranteed to induce policies that are Pareto-optimal according to our preference relation. Finally, we empirically evaluate tiered reward functions on several environments and show they induce desired behavior and lead to fast learning.

Estimating the probability of failure for complex real-world systems using high-fidelity computational models is often prohibitively expensive, especially when the probability is small. Exploiting low-fidelity models can make this process more feasible, but merging information from multiple low-fidelity and high-fidelity models poses several challenges. This paper presents a robust multi-fidelity surrogate modeling strategy in which the multi-fidelity surrogate is assembled using an active learning strategy using an on-the-fly model adequacy assessment set within a subset simulation framework for efficient reliability analysis. The multi-fidelity surrogate is assembled by first applying a Gaussian process correction to each low-fidelity model and assigning a model probability based on the model's local predictive accuracy and cost. Three strategies are proposed to fuse these individual surrogates into an overall surrogate model based on model averaging and deterministic/stochastic model selection. The strategies also dictate which model evaluations are necessary. No assumptions are made about the relationships between low-fidelity models, while the high-fidelity model is assumed to be the most accurate and most computationally expensive model. Through two analytical and two numerical case studies, including a case study evaluating the failure probability of Tristructural isotropic-coated (TRISO) nuclear fuels, the algorithm is shown to be highly accurate while drastically reducing the number of high-fidelity model calls (and hence computational cost).

Manually analyzing spermatozoa is a tremendous task for biologists due to the many fast-moving spermatozoa, causing inconsistencies in the quality of the assessments. Therefore, computer-assisted sperm analysis (CASA) has become a popular solution. Despite this, more data is needed to train supervised machine learning approaches in order to improve accuracy and reliability. In this regard, we provide a dataset called VISEM-Tracking with 20 video recordings of 30s of spermatozoa with manually annotated bounding-box coordinates and a set of sperm characteristics analyzed by experts in the domain. VISEM-Tracking is an extension of the previously published VISEM dataset. In addition to the annotated data, we provide unlabeled video clips for easy-to-use access and analysis of the data. As part of this paper, we present baseline sperm detection performances using the YOLOv5 deep learning model trained on the VISEM-Tracking dataset. As a result, the dataset can be used to train complex deep-learning models to analyze spermatozoa. The dataset is publicly available at https://zenodo.org/record/7293726.